The present invention relates to a method for analyzing motions of a human being or an animal and animating the results on a computer screen using a computer. The results of this method can be used, for examples to analyze a physical motion used in performing a sport (hereinafter referred to as a sport motion) in order to develop a curriculum for teaching the motion. It can also by used for analyzing various skills in the industrial and performing arts and forming curricula for teaching those skills. Such a method can also be used to train an animal, such as a dog or a horse, by analyzing the motions of the animal and deriving a training method.
In order to teach a new movement when performing a sport or a skill, such as in the industrial and performing arts or when training an animal or the like, it is necessary first to analyze the motions of the person or animal involved. In the past, a trained analyst observed the actual motions of a skilled athlete or an expert in the skill to be taught or a skilled animal, and then, on the basis of such observations, a curriculum or a program for teaching or reproducing the motions was formed. However, because such observations are subjective, the results may not be reliable. Furthermore, since the development of a new sport motion or skill is performed by trial and error, it can be very time consuming.
There has recently been proposed a technique wherein the motions of a human being or an animal are analyzed using a computer. See D. I. Miller "Computer Simulation of Human Motion", in D. W. Grieve et al (eds) Techniques for the Analysis of Human Motion, Lepus Books, London 1975; Calvert T. W. et al "Composition of Multiple Figure Sequences for Dance and Animation", Proceedings CG International '89, pp. 245-254 (1989); and Nigel W. J. et al, "Some Methods to Choreograph and Implement Motion in Computer Animation", Proceedings of Computer Animation '89, 125-140 (1989). In analyzing motion using a computer, a motion analyst, such as a teacher or a trainer, analyzes the motions depending on his understanding. Thus, the teacher or the trainer can use the computer analysis to teach an unskilled person or animal, the motion being studied so that such person or animal can perform the skill involved. Such analysis can also be used to develop new motions. Such computer analysis utilizes kinematics, which describes the motions only in terms of positions, velocities and accelerations, neglecting the forces and torques responsible. Thus, kinematics analysis can only generate a two-dimensional line picture representing parts constituting a human or animal body, and a three-dimensional model of the human or animal body cannot be displayed realistically on a screen. Accordingly, it is difficult to understand the display and also it is difficult to develop a new motion using the display.
Furthermore, while computer analysis of motions for developing new motions and new skills preferably utilizes a method having real time response, conventional computer analysis methods have no real time response because they require the ascertainment of the contents of actual motions and fine adjustments based on the results.
Another method of motion analysis uses dynamics. While kinematics provides motion analysis in terms of positions, velocities and accelerations, dynamics analyzes the motions of objects based on the relation between movements and forces. If dynamics is applied to the computer analysis of actual motions, it is possible to generate complex behavioral or animated results with minimal control by the analyst. Furthermore, an animation method utilizing dynamics has the great advantage of avoiding the limitations of methods utilizing kinematics.
However, the computer analysis of motions utilizing dynamics requires data on dynamics parameters such as the moments of inertia, the centers of gravity, joint friction and muscle/ligament elasticity of the moving body being represented by the animation, which parameters are difficult to measure. Without such data, the computer analysis of motions based on dynamics provides unrealistic motions similar to the results produced by kinematic motion analysis. Furthermore, dynamics motion analysis requires the solution of rather complex dynamics equations. For a human body an articulated body with 200 degrees of freedom, it is necessary to solve 600 simultaneous differential equations.
Thus, motion analysis methods which have been proposed thus far are not well suited for representing complex motions. Therefore, there is no motion analysis method capable of representing all the motions of a human or animal.
Conventional computer motion analysis methods utilizing dynamics to represent the movements of a human or animal body, for example, involve the following steps:
(1) Constructing a model of the human or animal body; PA1 (2) Applying the actual motions of a human or animal to the model; PA1 (3) Analyzing the motions of the model; and PA1 (4) Reproducing the analyzed motions.
In the fourth step (reproducing the analyzed motions), a method which can exactly solve the dynamics equations using the Gibbs formula is particularly suitable. However, for an actual motion analysis system, this method has not been used because of its conceptual and computational complexity, since when n is the number of segments constituting the model and forming minimal units of motion, the computational complexity of O(f(n)) becomes a function of O(n.sup.4) of n.sup.4, and thus is very large. Thus, dynamical simulation methods such as that disclosed in Wilhelms J. P et al., "Using Dynamic Analysis to Animate Articulated Bodies such as Humans and Robots", Proceedings, Graphics Interface '85, 97-104 (1985) have not been accepted, mostly because of its conceptual complexity and computational expensiveness.
A motion analyzing method which reduces the computation complexity to O(n), a function of n, enabled by neglecting rotations about the principal axes, has been proposed. Armstrong W. W. et al, "The Dynamics of Articulated Rigid Bodies for Purposes of Animation", The Visual Computer, 1:231-240 (1985). However, when the rotations of joints about the principal axes cannot be neglected, this method is not applicable. Thus, by using this method the forces that produce a motion or the forces that are produced by a motion cannot be displayed exactly and a realistic three-dimensional model of an articulated body cannot be obtained.
In a publication from the department of the present inventors, Sun, L. et al., "An Architectural Design of a Sports Instruction System", First International Conference and Exhibition on Visual Computing for Defense and Government (VISUDA '89), Paris, France (1989) (Technical Report 89-017), a system for inputting and analyzing human body movement is disclosed. The human body is modeled as a set of articulated segments, each of which is essentially a rigid solid. In this model, the human body is represented by 50 segments. The system takes in a set of measurements taken with an actual human body and converts it into the size data for each segment, e.g., length, width and diameter. When displaying the model, the system approximates each segment with a polygon and performs a smooth shading for better appearance. In addition the system calculates the center of gravity and the mass for each segment which will be required in the analysis. A video recorder, preferably several video recorders, recording the movement from a plurality of points of view, records the motion of actual humans in a time sequence of video frames. The position of objects in each frame is input to the system to reproduce and analyze the movement. In order to define the position and configuration of the body in space, the position and orientation of each segment are specified. Therefore, for each segment there are six degrees of freedom, three for the position and three for the orientation. Each segment in an articulated body, however, is restricted by attachments to neighboring segments. Therefore, it suffices to specify the position of only one segment (orientation must be specified for all segments), thereby reducing the degrees of freedom considerably. The user specifies the position of each body segment by manipulating this figure using a mouse; the user picks a segment and translates and or rotates it so that the displayed figure assumes the position of the figure on the videotape. The image of a video frame can be superimposed on each window to facilitate this process.
The body movement which has been input into the computer is then analyzed considering two factors: the movement of the center of gravity and the force exerted on each segment of the body. The position vector G of the center of gravity of a human body is computed by the following formulation: ##EQU1## where g.sub.i is the position vector of the center of gravity of a segment i and .DELTA.m.sub.i in its mass. Since each segment is assumed to be a uniform rigid body, g.sub.i and .DELTA.m.sub.i can be obtained prior to this calculation from the measurement data.
The center of gravity of each segment is computed in a way similar to the above formulation. Each segment is divided into small wedge-shaped volumes around and along the central axis through the segment, and these volumes are summed up to find the position of the center of gravity.
The problem of computing the force exerted on a human body from its movement is a problem of inverse dynamics. One starts with the sequence of positions and obtains the velocity and the acceleration of each part of the body, from which the force that has caused such motion is to be calculated.
Solving a problem of dynamics is essentially solving differential equations which describe the relationship between mass and force and torque applied to it. A variety of formulations are available for describing the dynamics equation. They all produce the same result expressed in slightly different terms. One of the most familiar formulations is Lagrange's equation: ##EQU2## where L is the Lagrangian given by kinetic energy (T) minus potential energy (P), q.sub.i is the ith coordinate, f is a constraint equation, .lambda. is an undetermined multiplier, and Q.sub.i is the force or torque applied to the ith coordinate.
In a general animation system, the dynamics approach is not yet practical due to the high computational complexity and cost required. The articulated body of the Sun publication model, consisting of 50 segments, requires 300 (=50.times.6) simultaneous equations to describe it. However, the use of known forces, such as the reactive force between a foot and a floor, simplifies the situation. For each segment, we have ##EQU3## where m is the mass of the segment, V.sub.g is the velocity vector of the center of gravity, X.sub.a and X.sub.b are the position vectors of its two ends, a and b, X.sub.a and X.sub.b are the velocity vectors, I is the inertia tensor, .omega..sub.g is he angular velocity vector around the center of gravity, and l is the distance between two ends,
Sun proposes letting the left side of he above Lagrange's equation be D. Then, EQU D(X.sub.a,X.sub.b,X.sub.a,X.sub.b,.lambda.F.sub.a,F.sub.b)=0
where F.sub.a and F.sub.b are the forces exerted on a and b, respectively. Each segment has six degrees of freedom, and there are six degrees of freedom, and there are six Lagrange's equations describing it. In the above Lagrange's equations D (X.sub.a,X.sub.b,X.sub.a, X.sub.b,.lambda.,F.sub.a,F.sub.b), the three components of F.sub.a (x,y,z) and F.sub.b (x,y,z), and .lambda. are unknown. Therefore, there are seven unknown variables in total in the the system of six equations. If the force in one direction can be obtained through some means (e.g. by measurement), .lambda. and five other force components can be computed by solving these equations.
Sun then supposes that a segment i is connected at an end b.sub.1 to an end a.sub.j of another segment j. By Newton's Third Law of Motion, the force exerted on the segment i at b.sub.i is equal and opposite to the force exerted on the segment j at a.sub.j. By solving the equations for the segment i, one also obtains a partial solution (three of the seven unknown variables) for the segment j as well. By repeating this process, solutions can be found for all segments making up the body.
As the starting point of a series of calculations, Sun suggests using the force exerted on a foot by the floor, which was measured in advance, to calculate the force exerted on the joint of the foot and shin. Then one works upward to head and lower arms through shins, thighs, hip waist and torso. In the end, one has calculated the force exerted on each segment at its ends. Note that the force calculated here is the sum of internal force and external force. Internal force is exerted by muscles on attached joints. For the segments on which no external force is exerted, the result is equal to just the internal force. On the other hand, for the segments with no internal force, the result is equal to the external force.
While this system of inputting and analyzing human body movement is useful for computerizing the dynamics of videotaped human movement, it does not solve the problem of how to produce practical animation works suitable for motion analysis and skill training, in a manner which simulates the motion of a human body as closely as possible, with a real time response, without relying on trial and error or an unworkable magnitude of calculations.